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On May 11, 2009, Space Shuttle Atlantis roared off of Launch Pad 39 A enroute to 
the Hubble Space Telescope (HST) to undertake its final servicing of HST, Servic- 
ing Mission 4. Onboard Atlantis was a small payload called the Relative Navigation 
Sensor experiment, which included three cameras of varying focal ranges, avion- 
ics to record images and estimate, in real time, the relative position and attitude 
(aka ’’pose”) of the telescope during rendezvous and deploy. The avionics package, 
known as SpaceCube and developed at the Goddard Space Flight Center, performed 
image processing using field programmable gate arrays to accelerate this process, 
and in addition executed two different pose algorithms in parallel, the Goddard Nat- 
ural Feature Image Recognition and the ULTOR Passive Pose and Position Engine 
(P3E) algorithms. 


INTRODUCTION 

In May of 2009, the National Aeronautics and Space Administration (NASA) Goddard Space 
Flight Center (GSFC) completed a successful on-orbit demonstration of its Relative Navigation 
Sensor (RNS) system. Flown in the cargo bay of Space Shuttle Atlantis during the highly success- 
ful Hubble Space Telescope (HST) Servicing Mission 4 (SM4), RNS captured and stored several 
hours of imagery of the telescope, as well as raw Global Positioning System (GPS) data, during the 
Rendezvous Proximity Operations and Docking (RPOD) and Deploy phases of the mission. The 
system also processed images of Hubble in real-time, estimating the position and orientation, or 
“pose” of the telescope relative to RNS’s Shuttle-mounted cameras. 

The RNS experiment on SM4 was originally conceived of in the course of developing a prelimi- 
nary mission concept and spacecraft design for the Hubble Space Telescope Robotic Servicing and 
De-orbit Mission (HRSDM). Intended to successfully rendezvous and dock with an uncontrolled 
HST, the servicing vehicle required an autonomous means for real-time estimation of the relative 
position and attitude of HST. We refer to this relative state measurement, made directly from sensor 
data without consideration of the system dynamics, as the “pose” measurement. 

The RNS experiment leveraged hardware procured prior to the cancellation of HRSDM to as- 
semble a relative navigation sensor payload, RNS, to be installed in the Space Shuttle payload bay 
during SM4. Hosted on the Multi-use Logistic Equipment Carrier (MULE), RNS made real-time 
pose measurements during approach to and departure from HST during SM4. The objectives of 
RNS were: 1) to record images of HST, especially of a new Soft Capture Mechanism (SCM) being 
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installed during SM4 to facilitate future RPOD with the Telescope; 2) to demonstrate the feasibility 
of generating real-time, on-board pose estimates under orbital lighting conditions; 3) to evaluate the 
performance of a comprehensive pose estimation system on orbit to assess its adequacy for possi- 
ble future Autonomous Rendezvous and Docking (AR&D) operations with an uncontrolled HST. 
These objectives were subject to limitations, including that HST remained controlled during the 
entire SM4, and RNS could neither specify nor control the Shuttle trajectory or attitude relative to 
the HST. 

RNS system hardware includes 3 cameras with varying optical ranges, a Navigator GPS receiver, 
an electronics box containing several commercial hard drives comprising the Mass Storage Module 
(MSM) to record camera images and raw GPS data, and an advanced microprocessor system called 
SpaceCube. Two pose estimation applications, the Goddard Natural Feature Image Recognition 
(GNFIR) and the ULTOR Passive Pose and Position Engine (P3E) (ULTOR), were executed in the 
SpaceCube. The primary requirements of these algorithms were to share the available processing 
resources — a Xilinx Virtex 4, each — and to produce pose estimates from each available camera 
(within their appropriate ranges) with sufficient precision to support evaluation of the accuracy 
goals stated in Table 1. 


Table 1 RNS pose accuracy goals (3cr) 


Target Lateral Range Roll Yaw/Pitch 

Range Accuracy Accuracy Accuracy Accuracy 

ft (m) ft (m) ft (m) 


492 

(150) 

3.3 (1) 

3.3 (1) 

15° 

15° 

98 

(30) 

0.9 (0.3) 

1.6 (0.5) 

5° 

5° 

16.5 

(5) 

0.3 (0.1) 

0.3 (.1) 

1° 

5° 


NOTE: Goals expressed in individual camera frames 


The focus of this paper is on pose estimation results, and on the comparisons of GNFIR and 
ULTOR algorithms to the best estimated trajectory provided by the Shuttle program. We first present 
a summary of the SM4 flight operations as they pertain to RNS, followed by additional details on 
the pose algorithms, and the first published pose estimation accuracy results from the SM4 flight. 
Details of the RNS sensor and avionics systems are described in detail in Ref. [1]. 

Flight Operations 

RNS flight operations were conducted in the Payload Operations Control Center (POCC) at John- 
son Space Center (JSC). While the prime method of commanding was via ground command, some 
commanding of the system could be performed by the crew. As described below, this backup mode 
of operation was critical during off-nominal operations in the Rendezvous phase of the mission. 
RNS flight operations consisted of the following major phases: Pre-Launch Operations; Activation; 
On-Orbit Checkout; Rendezvous; Post-Rendezvous; Deploy; Post-Deploy; De- Activation. 

Pre-Launch Operations included pre-launch checkout of the RNS Ground Terminal (RGT) in 
the JSC POCC, performed by routing simulated RNS data (compressed imagery via Ku-band and 
telemetry data and RNS commanding via S-band) to the two RGTs. 
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Figure 1 Rendezvous events timeline 

Activation of the RNS Telemetry Module (TM) was perfomed at Mission Elapsed Time (MET) 
0/02:33* (11 May 2009, 20:35 GMT). The TM remained powered for the remainder of the RNS 
operations, and provided health and safety telemetry during periods when SpaceCube-the primary 
source of RNS telemetry-was de-activated. 

On-Orbit Checkout commenced at MET 0/14:17 (12 May 2009, 08:19 GMT) via ground com- 
mand from the RGT. Checkout included the first power-up of the RNS sensors^ and avionics on orbit 
for verification. Proper operation of the system was verified when imagery of Earth, first recorded, 
and then live, were transmitted from the RNS system to the RGT via the Shuttle’s Ku-band link. 
Power switching and sensor data recording via Atlantis Crew command were also verified at this 
point. 

Rendezvous Operations commenced with power-up of the RNS operational heaters at MET 
1/19:40 (13 May 2009, 13:42 GMT) in preparation for rendezvous recording and pose estimation 
throughout the RPOD phase. Figure 1 shows the sequence of events during Rendezvous. 

RNS imaging operations during the Rendezvous phase commenced just after the Shuttle MC4 
burn - the final mid-course correction which targets the “R-bar” (HST nadir axis, as shown in Fig- 
ure 2). At this point in the RPOD phase, the Atlantis crew had already commanded the Shuttle to a 
“target track” attitude, with the HST-target at approximately the 1000 ft range and positioned above 
the Shuttle payload bay as shown in Figure 2 (the original figure, and more thorough description of 
the RPOD phase are available in Ref. [2]). HST was operational, in the “rendezvous” attitude as 
shown in Figure 2, with Solar “Beta” angle of approximately —10°. 

At this point in the rendezvous sequence RNS heaters were bringing the system up to its opera- 
tional temperature, the vehicles were passing through orbit night, and the ground team was debug- 
ging an issue with the Shuttle-HST S-band link, through which they needed to command HST to its 
final grapple configuration. After several other attempts failed, the ground team decided to switch 
over commanding of the Shuttle payloads to the backup hardware. While this did not resolve the 

*MET, or Mission Elapsed Time format is D/HH:MM:SS. MET 0/00:00:00 corresponds to HST SM4 liftoff 
at May 11, 2009 18:01:56 GMT. Detailed information on HST commanding during the servicing mission are available 
in the As Executed versions of the “Servicing Mission 4 Command Plan” and “HST Servicing Mission 4 Integrated 
Timeline (SMIT)”. 

^RNS sensors include 3 cameras, and the Navigator GPS receiver, always powered on and off simultaneously. 
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Figure 2 Shuttle R-Bar approach to HST and planned HST “rendezvous” and “grap- 
ple” attitudes (note that HST remained in the “rendezvous” attitude throughout the 
sequence, never maneuvering to the “grapple” attitude) 

issue, it did remove any possibility of RNS ground commanding for the remainder of the sequence 
(RNS ground commanding was limited to the primary communication hardware), t 

A second consequence of the Shuttle-HST link issue was that HST ground controllers in the Space 
Telescope Operations Control Center (STOCC) opted to forgo the HST roll maneuver to the “grap- 
ple” attitude. This misalignment would be removed later in the sequence by a crew-commanded 
Shuttle yaw maneuver of about 45° about the Payload Bay (Shuttle zenith) axis, performed at an 
inter- vehicle range of approximately 150 ft at MET 1/22:51 (13 May 2009, 16:53 GMT). This 
change to the relative approach attitude resulted in RNS being mis-configured for the initial pose ac- 
quisition. Ground commanding at this point could have adjusted the RNS pose algorithm acquisition 
modes. Unfortunately RNS commanding was not possible at this point due to the aforementioned 
command string switch over. 

Upon notification that command switch over would occur, the RNS ground team powered up the 
system and executed those commands which could not be performed by the Crew (including initia- 
tion of compressed image recording on the SpaceCube Video Interface Module (VIM)§). Recording 
of uncompressed imagery and GPS recording would be initiated at the originally planned time via 
Crew command which was still possible after the command string switchover. 

The remainder of the RPOD phase was nominal. The HST remained in an inertial hold through- 
out the sequence as the Atlantis Crew completed the “R-Bar” approach. The crew performed the 
yaw maneuver at approximately a range of 150 ft to align for grapple, transitioned the Shuttle’s 
attitude control system to inertial hold at an inter- vehicle range of approximately 130 ft to match 
rates with HST, and performed the final grapple and berth of the telescope using the Shuttle Remote 

*The issue was later resolved by reconfiguration of the link parameters on the HST side and RNS commanding was 
restored. 

§ The YIM recording script commenced almost an hour earlier than planned. The script was to cycle through cameras 
in an open loop fashion, recording compressed imagery at the times when HST was expected to be in their field of view. 
Because the script started early, very little of the recorded imagery shows HST. One benefit of this timing change is that 
YIM-recorded imagery, intended to be a redundant backup to MSM-recorded imagery, is unique. 
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Manipulator System (SRMS). As with previous HST servicing missions, the grapple occurred well 
after sunset, and RNS imagery of the final 130 ft of the approach is quite dark, since the Shuttle 
floodlights were the only source of illumination during orbit night. 

RNS rendezvous operations ceased with a crew-commanded stop recording and deactivation of 
the system at approximately 18:30 GMT. 



Figure 3 Image of Earth taken by RNS2 during RNS checkout (left), and of HST 
taken by RNS1 during rendezvous (right) 


Post-Rendezvous operations consisted of Ku-band downlink of the rendezvous imagery, and com- 
manding to prepare the RNS system for Deploy operations. In the days between Rendezvous and 
Deploy operations, the RNS team downlinked compressed imagery from all 3 cameras from the 
MSM using MSM playback, VIM compression, and SpaceCube transmission to the Shuttle Ku- 
band downlink system. 

Ku downloads of recorded rendezvous images occurred on Flight Days (FD) 3, 4, 5, 6, 7 and 
8. As a result of the downloads, approximately 81.5% of the recorded rendezvous images were 
downlinked for evaluation of camera Automatic Gain Control (AGC) performance. In total, 62,480 
unique images were downlinked over the course of 6 nights, representing almost 6 hours of unique 
camera footage. 

Evaluation of rendezvous imagery confirmed adequate performance with no updates to the cam- 
era AGC parameters necessary. Onboard storage space was confirmed to be adequate for deploy, the 
pose algorithms were commanded to Deploy Mode, and the system was ready for deploy operations. 

Deploy Operations were a critical time for RNS for it was this time to meet its major requirement 
of imaging of the new SCM installed on the aft bulkhead of HST. To properly image the SCM, 
the RNS Intermediate Position (RIP) was added into the timeline as part of the nominal mission 
plan. Therefore, deploy operations began at MET 07/14:44 (19 May 2009, 08:46 GMT), ahead 
of the scheduled un-berthing of HST. Operational heaters and the SpaceCube were powered on 
via commands from the RGT. The sensors and MSMs were powered on at MET 07/15:48 (09:50 
GMT), also via RGT command. Recording for deploy was initiated at MET 07/16:19 (10:21 GMT) 
via RGT command. The payload bay floodlights were turned on at MET 07/17:18 (11:20 GMT). 
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Figure 4 Image of HST Soft Capture Mechanism (SCM) taken by RNS3 during 
deploy, with SRMS at the RIP position (left) and by RNS2 after HST release (right) 

Deploy operations events relevant to RNS are shown in Figure 5. The deployment sequence pro- 
gressed as follows: Atlantis crew grappled HST with the SRMS; the HST ground team commanded 
de-mating of the HST umbilical, and opening of the three HST berthing latches on the Shuttle- 
mounted Flight Support Structure (FSS); the Atlantis crew then used the SRMS to maneuver the 
Telescope through a series of waypoints (see Figure 6), including the “FSS Hover” position, the 
“RIP” position, which centered the HST SCM in the field of view of the RNS short range camera, 
and the “HST Release Position”. Once the HST ground team had opened the HST aperture door, 
and deployed the HST high gain antennas, the crew released the telescope at MET 7/19:02 (13:04 
GMT), and performed two small separation burns to safely depart the neighborhood of HST. The 
RNS system continued to record during the final Shuttle-HST deployment. 


Orbit Day 
RNS Sensors Powered 
MSM GPS Recording 
MSM Cam Recording 
Shuttle Free Drift 
HST Unberth 
SRMS @ FSS Hover 
Ground GNFIR Track 
Fit GNFIR Track 
SRMS @ RIP 
SRMS @ HST Release Pos. 

HST AD to OPEN 
HST Release 
Shuttle SEP1 Burn 
Shuttle SEP2 Burn 



10:00 11:00 12:00 13:00 14:00 15:00 

GMT (starting on 05/19/2009) 


Figure 5 Deploy events timeline 

At MET 07/19:46 (13:48 GMT), all camera hard drives were full, but the MSM continued to 
record GPS data. At MET 07/21:06, all recording was stopped. The sensors and MSMs were 
powered off at MET 07/21:07. The SpaceCube was left powered for the next 24 hours to perform 
radiation mitigation testing. 
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Figure 6 Port views of Shuttle and HST : (top row) Grapple (left), FSS Hover (center), 
and FSS Berth (right); (bottom row) RIP (left) and HST Deploy (right) 


Post-Deploy Operations for downlink of deploy imagery and extended SpaceCube operations 
resumed during crew sleep after HST deploy. Ku downloads of recorded deploy images occurred 
during only one planning shift split between Flight Days (FD) 9 and 10. This was due to the fact 
that the majority of the remaining Shuttle payload was deactivated to conserve power for extended 
Shuttle operations. Mission managers made this decision on Flight Day 10 as forward forecasts of 
weather conditions at Kennedy Space Center (KSC) looked unfavorable for an on-time landing. As 
a result of this single night of downloads, approximately 15.3% of the recorded Deploy images were 
acquired.^ 

In total, 16,990 unique images were downloaded of the HST deployment sequence for more than 
an hour and a half of unique camera footage. 

De-Activation of the RNS TM occurred earlier than originally planned, at MET 8/21:12:53 to 
conserve energy onboard Shuttle. This was necessary due to inclement weather at the primary 
landing sight (KSC). 

ALGORITHMS 

GNFIR 

The GNFIR is the latest mutation of RAPiD 3 - one of the first 3D trackers to run in real-time. As 
was the case with RAPiD and its numerous descendants, GNFIR utilizes natural features (edges) 
on its target and requires neither fiducials nor a cooperative target. GNFIR is built upon a RAPiD- 
derived edge-tracker re-parameterized using a Lie group formalism developed by Drummond and 
Cipolla 4 at the University of Cambridge. This approach assumes a monocular (single camera) 
system and requires an internal three-dimensional stick-model of the edge features of interest. 

^All Rendezvous and Deploy images are currently available in uncompressed format: they have been recovered from 
the MSMs during post-flight operations at GSFC. 
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As with all model-based methods, a correspondence is made between an image pixel and a 3D 
model point and the projection transformation between them is estimated. A component of that 
3D to 2D transformation is the object’s “pose” relative to the camera frame. For GNFIR, pose is 
defined as the three elements to describe the position, and four quaternion elements to parameterize 
the attitude, with q 4 as the scalar element. 
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Figure 7 Object Space to Image Space Transformation 


Homogeneous coordinates and transformations are used throughout to account for the rotational 
and translational components of the pose. For this application, the Camera Matrix, shown in Fig- 
ure 7, is assumed to be a known, fixed quantity. The flight cameras were rigorously tested to 
determine these parameters so that the only unknown is the Pose Matrix. 

The pose estimation process, depicted in Figure 8, begins with a captured 10-bit camera image 
that is then processed using a standard edge-enhancing image (Sobel) filter. To realize realistic 
processing rates, this operation was moved into FPGA firmware which enabled increased the pro- 
cessing rate capability of the system from 0.5 Hz to 3 Hz. This enabled GNFIR to keep up with the 
RNS camera frame rate while in tracking mode. 

Once the edge image has been generated, the tracking loop begins with an initial estimate of the 
target pose. This initial estimate, in the case of GNFIR, is generated by its internal acquisition mode. 
The GNFIR acquisition mode relies on scoring many stored (mission specific) pose estimates that 
were generated a priori. Therefore, it was imperative to have a coarse 1 1 prediction of the relative 
motion of the Shuttle and HST during the rendezvous and deploy sequences to properly generate 
these initial pose estimates. 5 In addition to utilizing its own initial pose estimates, GNFIR was 
capable of utilizing pose seeds from ULTOR (or the ground) to aid in its attempt to acquire. 

Given an initial pose estimate, the algorithm projects the wire-frame model edges into image 
space where they are dynamically segmented into control points. Next, the algorithm attempts to 
find edges in the actual camera image that match the control points and model edges. The edge 
search is a one dimensional search, normal to the projected edge in image space. Errors are then 
generated between the located edges and the edges in the image. The errors are then minimized in 
a least-squares fashion. The pose correction matrix is formed using a linearized attitude parame- 
terization and then used to update the current pose estimate, and the process repeats for subsequent 
image frames. 

If, at any time during tracking, the quality** metric falls below an established threshold, a re- 
acquisition is triggered. 

^Coarse trajectory design for acquisition is defined as 10° in attitude and 10 % of range 
** GNFIR defines quality as a ratio of the features found vs the sought features 
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Figure 8 GNFIR Pose Process Loop 
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ULTOR® 

Advanced Optical Systems (AOS) designed the ULTOR P3E (ULTOR) 6 to provide 6DOF state 
estimation of objects. ULTOR exists as VHDL firmware and is designed to be reconfigurable and 
portable. During RNS, ULTOR processed imagery from the RNS cameras to produce real-time 
results at full camera frame rates. 

Simulated imagery of HST was developed to provide training data for ULTOR of specific features 
on HST in the camera field of view. The training data is processed to develop filters of the object to 
be measured. The ULTOR process is based on a spatial frequency analysis of a correlation between 
the filter and the actual object as viewed by the RNS camera. As shown in Figure 9, the relative 
positions of these features based on the HST mechanical model are used to evaluate the position 
and attitude of HST. An N-point perspective algorithm evaluates the spatial relationship of the 
estimated position features against the measured position to produce Six Degree of Freedom (6DOF) 
information of the object. 

ULTOR acquires targets based on a search of the filter database to find the closest position and 
attitude. Once acquired, ULTOR transitions into tracking mode and processes through the filter 
database basd on the motion of the object. The filter database was designed to support the spec- 
ified dispersion of trajectory angles based on the relative motion of the Shuttle and HST during 
rendezvous and deploy. 



Figure 9 Feature based ULTOR ! (ULTOR !)P3E overview 


TESTING 

Alluded to in the previous algorithm section, the pose algorithms place importance on synthetic 
image generation. Test imagery is critical to have as an input for flight code development and veri- 
fication, and to predict lighting of the target for proper model-feature selection. The pose algorithm 
performance is tightly coupled to the model features that are visible on the target, and therefore a 
system was developed to predict the on-orbit lighting conditions for the SM4 rendezvous and deploy 
trajectories. 

This system encompasses a tool to model and animate 3D geometry, Geomod, a physical illu- 
mination and rendering system, Phillum. Phillum is a stochastic path tracer that uses Monte Carlo 
importance sampling to follow a large number of incoming light rays from a detector grid, through 
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Figure 10 Geomod showing Phillum working on a SM4 Rendezvous Image 

lenses and into a 3D geometry scene. It then traces the paths of the lights through the scene to 
determine the exposure at the detector. It uses radiometric light transport, geometric optics and sta- 
tistically unbiased Monte Carlo integration. A screenshot of the application is shown in Figure 10. 

For RNS, there is also an Automatic Gain Control (AGC) algorithm that determines the camera 
gain and integration interval, controlling the exposure of the next image by evaluating the infor- 
mation content of the current image. This creates a complex system that, without proper ground 
testing, could easily result in over/under-exposed images. The RNS team had the freedom to choose 
parameters for the AGC algorithm, but these could not be changed once the rendezvous or deploy 
sequence had started. Therefore, it was imperative to determine these settings a priori and the only 
way to achieve this was through simulation of the high dynamic range imagery with Phillum. As 
a testament to the accuracy of Phillum and the simulated SM4 trajectory, a comparison between a 
flight image and a Phillum generated image is provided in Figure 1 1 . 



Figure 11 Comparison between Phillum synthetic image (left) and a Flight image from Rendezvous 
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POSE ESTIMATION FLIGHT RESULTS 


This section presents a summary of on-orbit results of the pose algorithms, comparisons to a truth 
solution, and details the data analysis of the various pose solutions. The RNS hardware performed 
admirably during the HST SM4 mission by recording much of the rendezvous and deploy imagery of 
HST and returning it safely to the ground. During the rendezvous operations the GNFIR algorithm 
was able to track HST through the RNS1 camera from a range of 97 m (318 ft) to a range of 45 m 
(148 ft) with a peak quality of 99.2% and maintained a continuous track for 20 minutes and 27 
seconds. This success on rendezvous was mirrored in the deploy sequence of the telescope as 
GNFIR was able to track HST in the RNS3 camera at the hover position for 3 minutes and 48 
seconds, and then at the RIP position for 1 1 minutes and 43 seconds with a peak quality of 87.1 %. 

Recall from the Rendezvous Operations section, that the RNS team was unable to issue a com- 
mand to the RNS system to change to a different operating mode that would have enabled better 
acquisition for the pose algorithms. In the case of ULTOR, this was critical as they were unable to 
track HST on-orbit. The appearance of HST in the RNS1 camera was at the edge of ULTOR’s train- 
ing data. ULTOR was able to produce several pose estimates with good confidence, but not enough 
to transition into tracking. ULTOR did pass an initial pose seed to GNFIR that enabled it to tran- 
sition to tracking mode though. However, using the flight imagery, ULTOR was able to generate a 
post-processed ground solution by placing the algorithm in the correct operating mode. Doing this 
resulted in tracking HST for 13 minutes and 33 seconds. 

The performance of the pose algorithms once the rendezvous took HST out of the field of view of 
RNS1 and into RNS2 was, unfortunately, not as successful. During rendezvous, GNFIR was only 
able to track for brief moments. The quality of the pose acquisition was enough to put the algorithm 
into tracking mode, but the acquisition time proved to be too long at 12 to 15 seconds to maintain 
tracking of HST as the relative range closed too quickly. Figure 12 shows the expected on-orbit 
lighting of HST on the left compared to the actual, on the right. 



Figure 12 RNS2 with Phillum synthetic imagery (left), and Flight image (right) on Rendezvous 

The combination of the variance in on-orbit lighting, the change in approach trajectory described 
in the Rendezvous Operations section, and the speed of the approach all combined to cause GNFIR 
to not acquire on-orbit. When the flight imagery was re-processed on the ground, the GNFIR 


12 



algorithm was able to acquire and track HST in RNS2 for 1 minute and 43 seconds with a peak 
quality of 67.8%. To achieve this the GNFIR acquisition was modified to acquire the different 
trajectory followed on rendezvous, as well as some minor model simplifications. This underscores 
the fact that much of the RNS2 imagery was quite dark during rendezvous. 

Finally, for RNS3, neither GNFIR nor ULTOR tracked HST during rendezvous when it was 
moved from the grapple position to the hover position above the FSS. This was due to a variance 
of 0.55 m and 3.6° in the expected motion of HST from the grapple position to the hover position. 
The features presented to the pose algorithms were similar to those expected, as shown in Figure 13. 
However, the differences in orientation and range were outside the expected search range and this 
was enough to cause GNFIR to not acquire HST during rendezvous. However, on deploy, GNFIR 
performed quite well by tracking HST at the FSS hover position and the RIP position for a total 
time of 15 minutes and 31 seconds with a peak quality of 87.1 %. 



Figure 13 RNS3 with Phillum synthetic imagery (left), and Flight image (right) on Rendezvous 

In light of these issues on-orbit, the RNS experiment met all of its objectives by: 1) imaging the 
newly installed SCM shown in Fig 4, 2) generating 32 minutes and 10 seconds of real-time pose 
estimates, 3) evaluating the performance of the pose estimation system (this paper). 

Flight Data Comparison 

The initial undulation of success is now tempered with comparisons of the pose flight data to inde- 
pendent data sources, namely the Shuttle program’s best estimated trajectory. The Shuttle program’s 
Relative Best Estimate Trajectory (RELBET), was produced by JSC by filtering several different 
data sources such as onboard state estimates, rendezvous radar, COAS, IMU, and star tracker mea- 
surements, and taking into account timeline events such as maneuvers and corrections. RELBET’s 
3 a accuracies at 300 m are 2 m radial, 25 m in-track, and 2 m cross-track which is further discussed 
in the Interface Control Document (ICD) in Ref. 7. Shuttle attitude data was taken directly from 
instrument measurements, with an accuracy of 0.2° when Inertial Measurement Unit (IMU) data is 
available, and 0.75° otherwise, as given in the PATH ICD in Ref. 8. HST attitude was provided by 
the HST operations team, and the published accuracy of their solution is 1 arcsec during rendezvous. 
During the deploy sequence, the joint angles of SRMS are used to reconstruct the relative position 
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of HST with respect to Shuttle, and eventually the RNS cameras. Accuracy of the joint encoders 
was not available at the time of publication. 

The positions and attitudes of HST and the Shuttle are used to create a pose solution in the camera 
frame. This pose solution is then compared to the on-orbit and ground processed pose solutions for 
GNFIR and ULTOR. We will see discrepancies between the pose generated from Shuttle and HST 
position/attitude information, and the pose solutions from the RNS algorithms. To resolve this 
discrepancy, the RNS flight imagery was used to justify the pose solution from the RNS algorithms 
as the correct solution. As a result, the position data of the GNFIR post-processed solution will be 
compared to the GNFIR flight solution and the ULTOR post-processed solution. The attitude data 
of the Shuttle and HST does provide a valid truth solution, and will be used in comparing to the 
pose solutions. 

The comparison data is presented in the RNS camera frame. The camera frame is defined with 
+X (Cni) to the right and -\-Y (C 712 ) up in the image plane, where n — 1, 2, 3 for RNS1, RNS2, and 
RNS3, respectively. +Z (Cn 3 ) completes the right handed triad by being into the camera boresight, 
thus making the — Z axis correspond roughly to range to the target. The origin is placed at the 
theoretical center of the camera lens to correspond to a pin-hole camera model. 

Figure 14(a) shows the different data sets of GNFIR(flight data and post-processed data), ULTOR 
post-processed data, and RELBET flight data. All pose solutions show the same trend for the ren- 
dezvous trajectory, however the RELBET solution shows a bias in all axes, most pronounced in the 
Z axis of the camera. Figures 15(a) and 15(b) detail the RELBET solution compared to the GNFIR 
flight solution. An ULTOR comparison is not possible as ULTOR did not track HST in flight, but 
did track HST when post-processing the flight imagery with the appropriate filters. The comparison 
plot and the RMS differences in Table 2 show a 20 m difference when compared to RELBET. How- 
ever, when the pose solutions are cross-compared (GNFIR vs ULTOR), the position differences 
drop dramatically to less than 0.5 m for RNS1 (see Table 1). In Figure 15(b), the comparison of 
attitude data does not exhibit the same issues as position data. In this case, the attitude differences 
are 2.0° per axis, for a magnitude of 2.65° for GNFIR. This on-orbit difference is much better than 
the desired error of 15° stated in Table 1. The bottom plot of Figure 15(b) shows the principal angle 
that is generated from the quaternion comparison of the two solutions, and the quality metric of the 
GNFIR flight data. The quality during this time is exclusively greater than 85%, and peaks at 99.2% 
for a principal angle difference exclusively less than 5°. 

Figure 14(b) shows the different pose solutions, as well as pose data derived from RMS joint 
angle flight data and joint angles from the PDRS checklist, 9 denoted as ’’published”. This plot 
demonstrates another error in the flight data for the RMS joint angles. While the pose from flight 
data, ground processing and predicted pose from the published joint angles all have differences in 
the centimeter range (Table 2), the comparison to RMS flight data shows a 0.75 m difference. The 
image in 14(d) shows the GNFIR post-processed pose solution overlayed on a flight image, which 
demonstrates, at least qualitatively, that a difference of this size cannot exist. Figures 15(c) and 
15(d) show the comparison of GNFIR flight data to RMS flight data, and the SRMS published joint 
angles. Figure 15(c) shows the position difference mostly in the RNS3 — Z axis. However, the 
comparison to the published joint angles show a difference of only 6 cm in range, which is well 
within the desired accuracy of 0.1m stated in Table 1. The RSS subplot in this figure shows the 
bias removed when comparing to the published joint angles. Figure 15(d) compares the attitude 
solutions and shows only minor differences in attitude. The total angular error for roll is less than 
1°, as given in Table 2, which is better than the desired 5° stated in Table 1. 
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(c) Rendezvous - RNS1 


(a) Rendezvous 



(b) Rendezvous 

Figure 14 Pose Flight Data plotted with Shuttle RELBET and SRMS data (a & b); 
GNFIR pose solution with Flight Images (c & d) 
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(a) Rendezvous Position 



(c) Deploy Position 



(b) Rendezvous Attitude (d) Dep i oy Attitude 

Figure 15 Pose Comparisons using GNFIR Flight data to : (a & b) RELBET, (c & d) 
RMS joint angles 


Truth Reconstruction Issues 
RELBET 

Several issues arise when comparing a direct relative measurement to a filtered solution com- 
prised of absolute and relative measurements as seen in the RELBET data. Notice that all relative 
trajectory comparisons to RELBET end at GMT 16:37:30 since at this point the Rendezvous Radar 
was configured for Ku-band communications and RELBET is no longer valid. Furthermore, the 
Rendezvous Radar was set to low power at 16:11:25 GMT which resulted in increased noise in the 
radar angles. The poor behavior of RELBET is illustrated in Figure 14(a) where RELBET and the 
GNFIR post-processed solution diverge at precisely the time at which the Ku-band is turned off. 
This can be seen in the Cli axis, and a few minutes earlier along the CI2 axis. The CI3 axis shows 
an approximate 20 m bias during the entire GNFIR tracking window. The rendezvous image shown 
in Figure 14(c) shows the GNFIRground solution overlayed on a flight image at 16:44:37 GMT. 
While errors on the order of a single-digit meter and degree may not be perceptible to the human 
eye in this image, the solution clearly does not contain an error of 20 m. 
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Table 2 Pose estimation differences from GNFIR and ULTOR solutions 


Description Position (meters) Attitude (degrees) 


(R=Rendezvous, D=Deploy) 

Range 

Lateral 

Roll 

Pitch 

Yaw 

(R) 

GNFIR Flight/RELBET 

22.2° 

4.60" 

\.22 b 

1.53* 

1.79* 

(R) 

GNFIR RNS1 Ground/RELBET 

22.5° 

4.62 a 

1.45* 

1.25* 

1.46* 

(R) 

GNFIR RNS2 Ground/RELBET 

- 

- 

1.20* 

8.69* 

10.2* 

(R) 

ULTOR Ground/RELBET 

22.T 

4.16° 

0.15 b 

2.49 b 

1.99* 

(R) 

ULTOR Ground/GNFIR Ground 

1.55 

0.30 

0.79 

2.80 

3.12 

(R) 

GNFIR Flight/GNFIR Ground 

0.43 

0.063 

0.19 

0.81 

0.57 

(D) 

GNFIR Flight/SRMS 

0.75° 

0.24° 

0.74° 

0.92° 

0.9V 

(D) 

GNFIR Ground/SRMS 

0.76° 

0.23 a 

0.75" 

0.99" 

0.93 a 

(D) 

ULTOR Ground/SRMS 

0.64° 

0.18° 

3.99° 

2. IT 

4.85° 

(D) 

GNFIR Flight/SRMS Published 

0.061 

0.17 

0.84 

1.33 

0.74 

(D) 

GNFIR Ground/SRMS Published 

0.078 

0.18 

0.82 

1.43 

0.74 

(D) 

ULTOR Ground/SRMS Published 

0.048 

0.091 

3.42 

3.02 

4.2 

(D) 

ULTOR Ground/GNFIR Ground 

0.10 

0.19 

5.40 

1.89 

3.90 

(D) 

GNFIR Flight/GNFIR Ground 

0.015 

0.022 

0.60 

0.40 

0.30 


fl RELBET position and as flown SRMS-derived position and attitude are biased, 
as discussed in Truth Reconstruction Issues 

^RELBET Attitude is valid (derived directly from Shuttle and HST onboard attitude solutions) 


SRMS Joint Angles 

A first glance at the deploy results shows GNFIR’s pose estimate compared to pose reconstructed 
using SRMS joint angles with an error of 0.9 m. However, when comparing the Mission Evalua- 
tion Workstation Software (MEWS) SRMS joint angles to published SRMS joint angles in Ref 9, 
specifically at berth, a difference of 1-3° per joint was discovered. At berth, HST is physically con- 
strained to the Shuttle so a discrepancy this size cannot be ignored. A 2° error on a 7 m arm length 
causes 0.25 m of error at the end of the arm. The discrepancy is illustrated in Figure 14(b) where a 
plot of the published SRMS joint angles lies near the GNFIR solution, and the MEWS SRMS data 
exhibits a bias. The SRMS published joint angles 9 placed HST directly on top of the latch and at 
the correct orientation, whereas the MEWS data places HST down into the latching mechanism by 
approximately 0.75 m. 

Pose Algorithm Post-Processing Improvements 

The post processed solutions for GNFIR and ULTOR sought to change as little as possible from 
the flight algorithms to generate comparison data. For either algorithm to successfully track HST 
during rendezvous, the algorithms had to be looking for it in the right place, as described in the 
Rendezvous Operations section. The changes to GNFIR to process rendezvous imagery in RNS1 
entailed nothing else aside from allowing the algorithm to determine a pose for each imaged. The 
result was slightly improved quality over the course of the rendezvous, and an extension of the 
tracking window to a total time of 27 minutes and 28 seconds. In the case of RNS2, the algorithm 
required minor modifications to the HST stick-model to increase the quality of the pose solution 

^Flight processing by GNFIR was slightly less than 3Hz 
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using available features, and a modification to the acquisition to compensate for the shifted ren- 
dezvous trajectory. With these modifications, the algorithm was able to track for 1 minute and 43 
seconds. No modifications were necessary to reprocess RNS3 imagery for deploy. The result was a 
tracking window of 18 minutes and 39 seconds. 

Regarding ULTOR, the AOS team discovered a software bug in the tracking mode that resulted 
in losing the track of HST midway through the RNS1 imagery on rendezvous. When corrected, UL- 
TOR was able to track HST using rendezvous imagery for 20 minutes and 20 seconds. For deploy, 
AOS discovered that features selected from the training data, and their subsequent filters, were not 
optimal given the size of HST in the FOV of RNS3. Most of the features selected were associated 
with the center docking target. The training data used to develop these filters did not adequately 
model the lighting conditions, reflectivity and contrast encountered on-orbit. This resulted in only 
two or three features providing the tracking information, and ULTOR requires at least four features 
to produce a 6DOF pose estimate. To resolve this, ULTOR was placed into a higher resolution win- 
dowing mode for enhancing features on the target. This mode was not included in the flight version 
of the software. To create a post-processed solution, AOS modified the training imagery to more 
accurately reflect the actual contrast of the SCM features and enabled the windowing mode. This 
resulted in a consistent track of HST for 10 minutes and 44 seconds at the RIP position using deploy 
imagery. 

Ground Results 

In light of the issues with the flight data provided by the Shuttle, we are unable to provide a 
truly independent measure of the relative position of the Shuttle and HST. We now utilize the 
GNFIR post-processed solution as a measurement of the relative position. Figure 16(a) shows a 
comparison of the GNFIR flight pose and the ULTOR post-processed pose solutions to the GNFIR 
post-processed solution during rendezvous. This data shows improvement in the position differ- 
ences. The RSS position error in Table 2 gives an error of 0.43 m in range for GNFIR and a 1.55 m 
error for ULTOR. Comparing these to the desired accuracy given in Table 1 of 1 m, GNFIR is bet- 
ter than the requirement, but ULTOR exceeds it. Figure 17(a) shows a comparison of the attitude 
solutions of the pose sources to the RELBET attitude solution using the principal angle between the 
two quaternion solutions. For rendezvous, we see for the GNFIR solutions, errors of 1.5° per axis 
from Table 2 and a total principal angle of 2.5°, and for the ULTOR solution, the accuracy resulted 
in 5° in roll and 2.5° for pitch/yaw from Table 2. Both solutions are much better than the desired 
accuracy of 15° stated in Table 1. 

RNS2 is not presented in these plots due to the short time tracking window, but results were 
tabulated against the Postflight Attitude and Trajectory History (PATH) attitude solution. These 
errors of 1.2° in roll and greater than 8° in pitch and yaw did not meet the desired accuracy of 5° in 
Table 1. This again is due to the poor on-orbit lighting conditions as shown in Figure 12. 

Figures 16(b) and 17(b) show comparisons of the deploy pose solutions. In the case of position, 
the GNFIR flight pose solution and the ULTOR pose solution are compared to the GNFIR post- 
processed solution. We see here that the error between the two GNFIR solutions is much smaller at 
5 cm or less, whereas the ULTOR solution shows errors of 0.2 m. When compared to the published 
joint angles, the position solutions for both GNFIR and ULTOR show differences less than 10 cm. 
These differences are on par if not better than the desired accuracy of 0.1 m stated in Table 1. In 
comparing attitude solutions in Figure 17(b), we see less than 1° error for both sets of GNFIR pose 
solutions, however ULTOR shows a 7° error from the reconstructed pose solution from SRMS joint 
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GMT Time [HH:MM] GMT Time [HH:MM] 

(a) Rendezvous (b) Deploy 

Figure 16 Position Comparison of GNFIR post-processed solution to GNFIR flight 
data and ULTOR post-processed solution 



(a) Rendezvous (b) Deploy 

Figure 17 Principal Angular Differences between Shuttle BET and GNFIR flight, 

GNFIR post-processed, and ULTOR post-processed 

angles. The GNFIR solution is well under the desired accuracy of 1° and 5° for roll and pitch/yaw, 
respectively, from Table 1, however ULTOR exceeds the desired with a 4° difference in roll and a 
6.14° difference in pitch/yaw. 

A final note regarding the results presented here regards the accuracy of these relative measure- 
ments. The alignment of the RNS system to the Shuttle Atlantis can only be determined to approx- 
imately a degree. The alignment error of the RNS system to the MULE is well documented and 
determined to be 0.05°. However, it was not possible to measure the alignment of the MULE to the 
Shuttle, but it is assumed to be on the order of one degree at worst. 

CONCLUSIONS 

The RNS system met its goal of imaging, recording and computing pose estimates of HST dur- 
ing its rendezvous with and deploy from Shuttle Atlantis for the SM4 mission. The GNFIR pose 
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algorithm tracked HST on rendezvous for 20 minutes and 27 seconds with a peak solution quality 
of 99.2 %. During the deploy sequence, GNFIR tracked HST for a total of 15 minutes and 31 sec- 
onds with a peak quality of 87.1 %. The inability of the ULTOR algorithm to track HST during 
rendezvous is linked to the RNS system not being configured for the change in the relative trajec- 
tory detailed in the Rendezvous Operations section. In the case of the deploy sequence, AOS’s 
oversight to not include the windowing mode in the ULTOR flight algorithm resulted in a lack of 
on-orbit tracking. However, in post-processing ULTOR was able to track HST for 20 minutes and 
20 seconds using rendezvous imagery, and 10 minutes and 44 seconds using deploy imagery. 

In determining the accuracy of the pose solutions, several issues arose in reconstructing truth 
data. The best estimated trajectory provided by the Shuttle program includes a 20 m error during 
rendezvous due to issues with the Rendezvous Radar, and a 0.75 m error during the deploy sequence 
from joint angle measurements of the SRMS. The attitude solutions using the PATH solution and 
SRMS joint angles offers a much better comparison. 

Therefore, to determine position accuracy, it was necessary to use a post-processed solution from 
the GNFIR algorithm. This comparison resulted in range errors during rendezvous of 0.43 m for 
GNFIR and 1.55 m for ULTOR. Note that the peak quality for the GNFIR post-processed solution 
was 99.2 %. These results were in the regime of the desired accuracy given in Table 1 of 1 m. In 
the case of attitude errors during rendezvous, comparisons to the PATH data show an error of 2.4° 
RSSfor GNFIR and an error of 3.27° RSS for ULTOR. These values are well below the desired 
accuracy of 15°. 

For the deploy comparison using RNS3, again the GNFIR post-processed solution was utilized 
to compare position estimates. Comparison to GNFIR flight data resulted in less than 5 cm error 
in position while ULTOR resulted in a 0.2 m error in position. The peak GNFIR quality for the 
post-processed solution was 87.1 %. Regarding attitude accuracy, the GNFIR solutions show a less 
than 1° principal angle error, while the ULTOR data shows a 7° error in the principal angle. This 
principal angle error is an error generated by comparing quaternions. The desired accuracy for the 
short range camera, RNS3, was given in Table 1 as 1° in roll and 5° in pitch/yaw. From Table 2, 
GNFIR meets these easily, however ULTOR, with errors larger than 5° Root Sum Squared (RSS) 
does not. 

There is no denying the overall success of the RNS experiment on STS- 125. When delving into 
the detailed comparison data, the success is somewhat tempered by the comparison to truth but 
was sufficient to meet the project’s stated goals (Table 1). The lessons learned on RNS are directly 
applicable to future AR&D scenarios and the collected data is a valuable asset for pose algorithm 
development. 
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ACRONYMS AND ABBREVIATIONS 

6DOF Six Degree of Freedom 

AR&D Autonomous Rendezvous and Docking 

AGC Automatic Gain Control 

AOS Advanced Optical Systems 

COAS Crewman Optical Alignment Sight 

FOV Field of View 

FSS Flight Support Structure 

Geomod Geometry Modeling Tool 

GMT Greenwich Mean Time 

GNFIR Goddard Natural Feature Image Recognition 
GPS Global Positioning System 
GSFC Goddard Space Flight Center 
HST Hubble Space Telescope 

HRSDM Hubble Space Telescope Robotic Servicing and De-orbit Mission 

ICD Interface Control Document 

IMU Inertial Measurement Unit 

JSC Johnson Space Center 

KSC Kennedy Space Center 

MEWS Mission Evaluation Workstation Software 

MET Mission Elapsed Time 

MULE Multi -use Logistic Equipment Carrier 

MSM Mass Storage Module 

NASA National Aeronautics and Space Administration 

P3E Passive Pose and Position Engine 

PATH Postflight Attitude and Trajectory History 

Ph ilium Physical Illumination 

POCC Payload Operations Control Center 

RELBET Relative Best Estimate Trajectory 
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RGT RNS Ground Terminal 
RIP RNS Intermediate Position 
RMS Root Mean Square 
RNS Relative Navigation Sensor 
RNS1 RNS Camera 1 (long range) 

RNS2 RNS Camera 2 (medium range) 

RNS3 RNS Camera 3 (short range) 

RPOD Rendezvous Proximity Operations and Docking 

RSS Root Sum Squared 

SCM Soft Capture Mechanism 

SM4 Servicing Mission 4 

SMIT Servicing Mission 4 Integrated Timeline 

SRMS Shuttle Remote Manipulator System 

STOCC Space Telescope Operations Control Center 

TM Telemetry Module 

ULTOR ULTORP3E 

VIM Video Interface Module 

VHDL VHSIC hardware description language 
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